Speech intelligibility derived from asynchronous processing of auditory-visual information

The current study examines the temporal parameters associated with cross-modal integration of auditory-visual information for sentential material (Harvard/IEEE sentences). The speech signal was filtered into 1/3-octave channels, all of which were discarded (in the primary experiment) save for a low-frequency (298-375 Hz) and a high-frequency (4762-6000 Hz) band. The intelligibility of this audio-only signal ranged (depending on the listener) between 9% and 31% (of the key words correct) for nine normal-hearing subjects. Visual-alone presentation of the same material ranged between 1% and 22% intelligi-bility. When the audio and video signals are combined and presented in perfect synchrony, intelligibility climbs to an average of 63%. The audio and video signals were systematically desynchronized in symmetrical fashion up to a maximum onset asynchrony of 400 ms. When the audio signal leads the video, intelligibility declines appreciably for even the shortest asynchrony of 40 ms, falling to an asymptotic level of performance for asynchronies of ca. 120 ms and longer for most subjects. In contrast, when the video signal leads the audio, intelligibility remains relatively stable for onset asynchronies up to 160-200 ms, and in some instances may actually improve. Hence, there is a marked asymmetry in the integration of audio and visual information that has important implications for sensory-based models of auditory-visual speech processing.